System and methods for reconstructing medical images using deep neural networks and recursive decimation of measurement data

ABSTRACT

Methods and systems are provided for reconstructing images from measurement data using one or more deep neural networks according to a decimation strategy. In one embodiment, a method for reconstructing an image using measurement data comprises, receiving measurement data acquired by an imaging device, selecting a decimation strategy, producing a reconstructed image from the measurement data using the decimation strategy and one or more deep neural networks, and displaying the reconstructed image via a display device. By decimating measurement data to form one or more decimated measurement data arrays, a computational complexity of mapping the measurement data to image data may be reduced from O(N4), where N is the size of the measurement data, to O(M4), where M is the size of an individual decimated measurement data array, wherein M&lt;N.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 16/691,430, entitled “SYSTEM AND METHODS FORRECONSTRUCTING MEDICAL IMAGES USING DEEP NEURAL NETWORKS AND RECURSIVEDECIMATION OF MEASUREMENT DATA”, and filed on Nov. 21, 2019. The entirecontents of the above-listed application are hereby incorporated byreference for all purposes.

TECHNICAL FIELD

Embodiments of the subject matter disclosed herein relate to medicalimaging, and more particularly, to systems and methods forreconstructing medical images using deep neural networks.

BACKGROUND

Medical imaging systems are often used to obtain internal physiologicalinformation of a subject, such as a patient. For example, a medicalimaging system may be used to obtain images of the bone structure, thebrain, the heart, the lungs, and various other features of a patient.Medical imaging systems may include magnetic resonance imaging (MRI)systems, computed tomography (CT) systems, positron emission tomography(PET) systems, hybrid PET/MR systems, x-ray systems, ultrasound systems,C-arm systems, and various other imaging modalities.

Measurement data obtained by a medical imaging system (e.g., x-rayprojection data in CT imaging, or k-space data in MRI), does not lenditself to visual analysis/diagnosis by a human, as the anatomicalinformation encoded therein may be in a form which is not easily orintuitively processed by the human brain. For example, in MRI imaging,measurement data, also referred to as K-space data, comprises a two orthree dimensional Fourier transform of image data, wherein each point ofK-space is related to the image intensity of every pixel/voxel of acorresponding image, and therefore K-space data may be prohibitivelydifficult for a human mind to relate to the underlying anatomicalfeatures encoded therein. Therefore, measurement data is conventionallyreconstructed to form medical images showing the anatomical structuresin a form more amenable to human inspection, enabling, amongst otherthings, diagnosis of the acquired medical images by a human expert.

Recently, machine learning approaches have been implemented to directlymap measurement data to medical image data, without relying onconventional approaches such as filtered backpropagation (FBP), homodynealgorithms, zero filling methods, dictionary learning, and, projectionsonto convex sets, etc. Machine learning approaches may enable more rapidmedical image reconstruction, enabling shorter scan times and/or smallerdoses of scanning radiation. However, the computational complexity (andtherefore the time/computational resources) required in currentapproaches to train and implement fully connected deep neural networkscapable of directly mapping measurement data to image data increases bythe fourth power of the matrix size of the medical image. This“explosion of parameters” prohibits implementation of such approaches incomputationally restricted environments, or for use with high-resolutionor three-dimensional (3D) medical images. Therefore, it is generallydesirable to explore techniques for reducing a computational complexityof medical image reconstruction using deep neural networks.

SUMMARY

In one embodiment, a method for reconstructing an image from measurementdata comprises, receiving measurement data acquired by an imagingdevice, selecting a decimation strategy, producing a reconstructed imagefrom the measurement data using the decimation strategy and one or moredeep neural networks, and displaying the reconstructed image via adisplay device. In this way, measurement data may bedownsampled/decimated according to a decimation strategy, therebysubstantially reducing the number of parameters of the one or more deepneural networks. By decimating measurement data to form one or moredecimated measurement data arrays, a computational complexity of mappingthe measurement data to image data may be reduced from O(N⁴), where N isthe size of the measurement data, to O(M⁴), where M is the size of anindividual decimated measurement data array, where M<N, and where M maybe selected independently of the image resolution.

The above advantages and other advantages, and features of the presentdescription will be readily apparent from the following DetailedDescription when taken alone or in connection with the accompanyingdrawings. It should be understood that the summary above is provided tointroduce in simplified form a selection of concepts that are furtherdescribed in the detailed description. It is not meant to identify keyor essential features of the claimed subject matter, the scope of whichis defined uniquely by the claims that follow the detailed description.Furthermore, the claimed subject matter is not limited toimplementations that solve any disadvantages noted above or in any partof this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of this disclosure may be better understood upon readingthe following detailed description and upon reference to the drawings inwhich:

FIG. 1 shows a block diagram of an exemplary embodiment of an imagereconstruction system;

FIG. 2 is a schematic diagram illustrating an imaging system capable ofimplementing the image reconstruction system of FIG. 1 ;

FIG. 3 is a flowchart illustrating a method for reconstructing a medicalimage from measurement data using one or more deep neural networksaccording to a decimation strategy;

FIG. 4 is a flowchart illustrating an exemplary method for training theone or more deep neural networks of FIG. 3 using training data pairscomprising measurement data and ground truth medical imagescorresponding to the measurement data; and

FIG. 5 shows a comparison between an image reconstructed according to anexemplary embodiment of the current disclosure, and a medical imagereconstructed using a conventional approach.

The drawings illustrate specific aspects of the described systems andmethods for reconstructing images using deep neural networks and arecursive decimation strategy. Together with the following description,the drawings demonstrate and explain the structures, methods, andprinciples described herein. In the drawings, the size of components maybe exaggerated or otherwise modified for clarity. Well-known structures,materials, or operations are not shown or described in detail to avoidobscuring aspects of the described components, systems and methods.

DETAILED DESCRIPTION

The following description relates to systems and methods forreconstructing medical images from their corresponding measurement databy recursive manifold approximations using neural networks. Thisapproach is applicable to image reconstruction where the measurementdata is related to a corresponding reconstructed image or correspondingoutcome through a linear transform (e.g., Fourier transform, Radontransform, etc.). Although discussed herein in the context of medicalimages, it will be appreciated that the current disclosure provides forimage reconstruction of substantially any imaging modality, and is notlimited to medical images. Existing neural network-based methods whichattempt to learn the high dimensional manifold relationship directlysuffer from explosion of the number of parameters of the neural networkswith increasing measurement data size, making such approachesimpractical for data sizes observed in practice. The current disclosureenables neural network based image reconstruction directly frommeasurement data, with a reduction in the number of neural networkparameters used/learned, by decimating the measurement data recursivelyto smaller sub-units (herein referred to as decimated measurement dataarrays) which are transformed into latent manifold embeddings (hereinreferred to as decimated image data arrays) through a first neuralnetwork. This is followed by recursive aggregation of the decimatedimage data arrays using an aggregation network, to produce areconstructed medical image. Both the first neural network and theaggregation network parameters are learned together in a joint-mannerusing supervised training.

The disclosed systems and methods enable a reduction in computationalcomplexity of mapping the measurement data to image data from O(N⁴), formeasurement data of size N×N, to O(M⁴), where M is the size of anindividual decimated measurement data array, M<N, and where M may beselected independently of N. This reduction in parameters does notcompromise the quality of image reconstruction, compared to conventionalreconstruction approaches. Additionally, as the neural networks taughtherein comprise substantially fewer parameters than conventional neuralnetworks, the current approach is resistant to over-fitting, and datamemorization, which are commonly cited as hindrances to adaptation ofdeep neural networks in image reconstruction.

The reduced computational complexity afforded by the current disclosureenables neural network based image reconstruction using measurement dataon large, three-dimensional (3D) volumes generated in MRI, CT and PETclinical scanners. The current disclosure enables a dimensionalityreduction of measurement data by decimating measurement data intosmaller, and/or lower dimensional order, arrays, which may then beprocessed by one or more smaller deep neural networks. The reducedmemory footprint of the one or more deep neural networks enables theapproach to be implemented on hardware compromised devices, such asmobile phones, or CPU only systems. Further, the approaches of thecurrent disclosure are applicable to a variety of imaging modalities,including magnetic resonance imaging (MRI), computed tomography (CT),positron emission tomography (PET), PET/MR, and C-Arm. The systems andmethods disclosed herein may enable deployment of machine learning basedimage reconstruction, without the need for specialized GPU hardware, onexisting computer platforms.

In one embodiment, the image reconstruction system 100, shown in FIG. 1, may be implemented by a medical imaging system 200, shown in FIG. 2 ,to reconstruct a medical image from measurement data according to one ormore of the operations of method 300, shown in FIG. 3 . Briefly, method300 comprises, receiving measurement data of an anatomical region of apatient, recursively decimating the measurement data according to adecimation strategy to form a plurality of decimated measurement dataarrays, mapping the plurality of decimated measurement data arrays to aplurality of decimated image data arrays using one or more trained deepneural networks, and recursively aggregating the plurality of decimatedimage data arrays, using one or more trained aggregation networks, toform a reconstructed medical image. FIG. 4 shows a method for trainingthe plurality of deep neural networks and the one or more aggregationnetworks employed in method 300, using a joint-training approach,wherein the parameters of the deep neural networks and aggregationnetworks are learned together, using a same training dataset. FIG. 5shows a comparison between a first medical image 502, which wasreconstructed from measurement data using the steps of method 300, witha second medical image 504, which was produced using a conventionalimage reconstruction approach.

Referring to FIG. 1 , an example image reconstruction system 100 isshown. Image reconstruction system 100 is one embodiment of an imagereconstruction system according to the current disclosure. Inparticular, image reconstruction system 100 implements one example of abinary decimation strategy, wherein measurement data 102 is dividedbetween a first decimated measurement data array 110 and a seconddecimated measurement data array 112. It will be appreciated that thecurrent disclosure provides for various decimation strategies, whereinmeasurement data may be sampled according to one or more samplingpatterns and sampling densities to produce substantially any number ofdecimated measurement data arrays. As an example, a trinary decimationstrategy may be employed, wherein measurement data array 106 isdivided/decimated into a first, second, and third decimated measurementdata array.

Further, measurement data 102 comprises two-dimensional (2D) k-spacedata (also referred to as a Fourier plane) corresponding toreconstructed medical image 128, which comprises a 2D MRI image slice ofhuman brain. However, it will be appreciated that the approach of thecurrent disclosure may be used to reconstruct various images of variousimaging modalities, including 3D images, CT images, PET images, etc.

Image reconstruction system 100 may receive measurement data 102 fromone or more communicably coupled devices, including non-transitory datastorage devices. In one embodiment, image reconstruction system 100receives measurement data 102 from an imaging device, such as imagingdevice 240, discussed in more detail below, with reference to FIG. 2 .In the embodiment shown in FIG. 1 , measurement data 102 comprises a128×128, 2D array of k-space data corresponding to a 2D medical image,wherein the spatial frequencies (and phases) of the intensitydistribution of the 2D medical image are encoded within measurement data102 as a Fourier plane. It will be appreciated that the currentdisclosure provides for reconstruction of medical images from varioustypes of measurement data, and of various dimensional order (including2D, 3D, etc.) wherein the measurement data is related to the medicalimage data via a linear transform.

Following receipt of measurement data 102 by image reconstruction system100, measurement data 102 is reformatted as a one-dimensional (1D),measurement data array 106, via a flattening operation, as indicated byflatten 104. Flatten 104 comprises vectorization of measurement data102, wherein the 2D arrangement of measurement data 102 is convertedinto an equivalent vector. In one embodiment shown by FIG. 1 ,measurement data array 106 comprises a 16,384 row vector of measurementdata, corresponding to measurement data 102. By flattening measurementdata 102, the data may be more easily decimated, as 2D, or 3D data maybe flattened to an equivalent 1D representation, wherein decimation maybe conducted in a more computationally efficient manner. As an example,a 2D sampling pattern may indicate a sampling frequency in both a firstdimension and a second dimension, wherein, a sampling pattern offlattened image data may express the 2D sampling pattern using a single1D sampling pattern. The computational savings may be even greater forthree-dimensional (3D) measurement data, where flatten 104 may reducethe dimensionality of the measurement data from 3D to 1D, enabling adecimation strategy comprising a 1D sampling pattern to be employed.

In some embodiments, additional data may be concatenated withmeasurement data array 106. In one example, in MRI, multi-coil data,comprising complex k-space data, may have an associated sensitivity map.The sensitivity map data may be flattened in a manner analogous to thatdescribed with respect to measurement data 102, and the flattenedsensitivity map data may be concatenated with measurement data array106. This will provide a concatenated vector of both k-space data andsensitivity map data. In other embodiments, such as in color imaging,measurement data 102 may comprise multiple color channels (e.g., RGBcolor channels, or CMYK color channels, wherein each channel comprisesintensity values corresponding to an associated color). The plurality ofintensity values for each associated color may be separately flattened,and then concatenated, to produce measurement data array 106.

The measurement data array 106 may then be decimated, as indicated bydecimation 108. Decimation 108 comprises dividing the measurement dataarray into two or more decimated measurement data arrays, according to adecimation strategy, wherein a decimation strategy indicates one or moreof a decimation factor, a sampling pattern, and a sampling density(which may be a function of the decimation factor). Regarding thedecimation factor, the decimation factor may comprise an integer or arational fraction greater than one. The decimation factor may be used todetermine a sampling rate or sampling interval/pattern. For example, ifmeasurement data array 106 comprises a 16,384 row vector, wherein eachrow includes a distinct intensity value from measurement data 102, andmeasurement data array 106 is decimated using a decimation factor of 4,four decimated measurement data arrays will be produced, each comprising4,096 rows of intensity values.

In the embodiment shown in FIG. 1 , the measurement data array 106 isdivided into a first decimated measurement data array 110, and a seconddecimated measurement data array 112, using a binary decimationstrategy. In one embodiment, a binary decimation strategy includes adecimation factor of 2, and a homogenous sampling pattern, which maycomprise apportioning each odd row of measurement data array 106 tofirst decimated measurement data array 110, and apportioning each evenrow of measurement data 106 to the second decimated measurement dataarray 112. In other embodiments, a heterogeneous sampling pattern may beemployed, wherein the sampling pattern varies as a function positionwithin measurement data array 106. It will be appreciated that a binary,trinary, etc., decimation strategy, producing two, three, etc.,decimated measurement data arrays, may each be obtained using aplurality of sampling patterns/sampling frequencies, and the currentdisclosure provides for each of the plurality of samplingpatterns/frequencies. As an example, although a binary decimationstrategy may comprise generating a first decimated measurement dataarray by selecting every-other intensity value in measurement data array106, other sampling patterns, such as selecting two consecutiveintensity values of measurement data array 106, followed by notselecting the next two consecutive values of measurement data array 106,may be employed. In another example, the decimation strategy maycomprise a heterogeneous sampling pattern, wherein a first samplingpattern is used for a first section of measurement data array 106, and asecond sampling pattern is used for a second section of measurement dataarray 106, etc.

First decimated measurement data array 110, and second decimatedmeasurement data array 112, may each comprise a same number ofentries/intensity values. The sum of the entries/intensity values infirst decimated measurement data array and second decimated measurementdata array may equal the number of entries/intensity values inmeasurement data 102 and/or measurement data array 106. In particular,first decimated measurement data array 110, and second decimatedmeasurement data array 112, each comprise 8,192 intensity valuesarranged into vectors of 8,192 rows.

The first decimated measurement data array 110 is received by an inputlayer of first network 114. In one embodiment, first network 114comprises one or more 2D convolutional layers implemented as denselayers, avoiding iteration over rows and columns by utilizing theparallelization of convolutional layers. Each of the dense/fullyconnected layers may be followed by a non-linear activation, using oneor more activation functions known in the art of machine learning. Inone example, a dense layer/fully connected layer may be implemented as a2D convolutional layer by setting the stride and receptive field suchthat each output of a current layer is mapped to each input of afollowing layer. The fully connected/dense layers may comprise one ormore learned filters, which map the Fourier plane intensity data offirst decimated measurement data array 110 to image intensity data offirst decimated image data array 118.

Similarly, the second decimated measurement data array 112 is receivedby an input layer of second network 116. In one embodiment, secondnetwork 116 comprises one or more 2D convolutional layers implemented asdense layers, thereby avoiding iteration over rows and columns byutilizing the parallelization of convolutional layers. Each of thedense/fully connected layers may be followed by a non-linear activation,using one or more activation functions known in the art of machinelearning. In one example, a dense layer/fully connected layer may beimplemented as a 2D convolutional layer by setting the stride andreceptive field such that each output of a previous layer is mapped toeach input of a following layer. The fully connected/dense layers maycomprise one or more learned filters, which map the Fourier planeintensity data of second decimated measurement data array 112 to imageintensity data of second decimated image data array 120. The one or morelearned filters of the one or more fully connected/dense layers of firstnetwork 114 and second network 116 may be learned via a training method,such as that discussed below with reference to FIG. 4 .

First decimated image data array 118, and second decimated image dataarray 120, may each comprise a same number of image intensity values. Inthe example shown in FIG. 1 , first decimated image data array 118 andsecond decimated image data array 120 each comprise 8,192 intensityvalues, corresponding to the 8,192 Fourier intensity values of firstdecimated measurement data array 110, and second decimated measurementdata array 112, respectively.

First decimated image data array 118 and second decimated image dataarray 120 are both input into an input layer of aggregation network 112.In one embodiment, first decimated image data array 118 and seconddecimated image data array 120, are concatenated row-wise, prior tobeing received by aggregation network 122. Aggregation network 122 maybe configured to receive multiple sources of input data. In one example,aggregation network 122 may be configured to receive image data from asame imaging modality (e.g., multi-coil MR data) or from differentimaging modalities (e.g., PET/CT data).

In the embodiment shown in FIG. 1 , aggregation network 112 is a 1Dconvolutional neural network, comprising one or more fullyconnected/dense layers, including one or more learned 1D convolutionalfilters. The 1D convolutional filters of aggregation network 122 may belearned during a training routine, such as the training routinedescribed below, with reference to FIG. 4 . The 1D fully connected/denselayers map the first decimated image data array 118 and the seconddecimated image data array 120, to image data array 124.

Image data array 124 comprises image intensity data, synthesized fromthe image intensity data of first decimated image data array 118 andsecond decimated image data array 120. In the example shown in FIG. 1 ,image data array 124 comprises a 16,384 row vector, corresponding to16,384 image intensity values.

Image data array 124 is re-shaped to produce reconstructed medical image128, as indicated by reshape 126. Reshape 126 comprises re-arranging therows of image data array 124 into a matrix. In one example, reshape 126comprises dividing image data array 124 at a plurality of pre-determinedrows, to produce a plurality of columns, and re-arranging the rows intoa pre-determined order, thereby producing a matrix having a number ofcolumns equal to the number of the plurality of columns.

Reconstructed medical image 128 comprises a medical image of a patient'sanatomy, corresponding to measurement data 102. In the embodiment shownin FIG. 1 , reconstructed medical image 126 comprises a 128×128 pixel,2D MR image of a patient's brain.

The image reconstruction system 100 of FIG. 1 enables rapid, efficient,and robust reconstruction of a medical image, directly from measurementdata, using one or more trained deep neural networks, while reducing acomputational complexity/number of learned parameters by decimating themeasurement data according to a pre-determined decimation strategy.

Referring to FIG. 2 , a medical imaging system 200 is shown, inaccordance with an exemplary embodiment. Medical imaging system 200comprises image processing system 202, display device 220, user inputdevice 230, and imaging device 240. In some embodiments, at least aportion of image processing system 202 is disposed at a device (e.g.,edge device, server, etc.) communicably coupled to the medical imagingsystem 200 via wired and/or wireless connections. In some embodiments,at least a portion of image processing system 202 is disposed at aseparate device (e.g., a workstation) which can receive images from themedical imaging system 200 from a storage device which stores the imagesgenerated by the medical imaging system 200.

Image processing system 202 includes a processor 204 configured toexecute machine readable instructions stored in non-transitory memory206. Processor 204 may be single core or multi-core, and the programsexecuted thereon may be configured for parallel or distributedprocessing. In some embodiments, the processor 204 may optionallyinclude individual components that are distributed throughout two ormore devices, which may be remotely located and/or configured forcoordinated processing. In some embodiments, one or more aspects of theprocessor 204 may be virtualized and executed by remotely-accessiblenetworked computing devices configured in a cloud computingconfiguration.

Non-transitory memory 206 may store deep neural network module 208,image reconstruction module 212, and image data 214. Deep neural networkmodule 208 may include one or more deep neural networks, comprising aplurality of weights and biases, activation functions, loss functions,and instructions for implementing the one or more deep neural networksto map measurement data to image data, and/or to map decimated imagedata to a single image data array. For example, deep neural networkmodule 208 may store instructions for implementing one or more neuralnetworks, according to one more steps of methods 300 and 400, discussedin more detail below. Deep neural network module 208 may include trainedand/or untrained neural networks and may further include various data,such as training data, training routines, or parameters (e.g., weightsand biases), associated with one or more neural network models storedtherein. Deep neural network module 208 may include instructions fortraining one or more of the deep neural networks. In one embodiment,deep neural network module 208 may include gradient descent algorithms,loss functions, and rules for generating and/or filtering training data.Deep neural network module 208 may include instructions that, whenexecuted by processor 204, cause image processing system 202 to conductone or more of the steps of method 400, discussed in more detail below.In one example, deep neural network module 208 includes instructions forreceiving training data pairs from image data 214, which comprise pairsof measurement data and corresponding ground truth medical images, foruse in training one or more of the deep neural networks stored in deepneural network module 208. In some embodiments, the deep neural networkmodule 208 is not disposed at the image processing system 202.

Deep neural network module 208 may include trained and/or untrainedneural networks and may further include various deep neural networkmetadata pertaining to the trained and/or untrained networks. In someembodiments, the deep neural network metadata may include an indicationof the training data used to train a trained deep neural network, atraining method employed to train a trained deep neural network, anaccuracy/validation score of a trained deep neural network, and a typeof imaging modality/imaging protocol for which the trained deep neuralnetwork may be applied.

Non-transitory memory 206 may further include image reconstructionmodule 212, which comprises instructions for implementing one or moredecimation strategies, in conjunction with one or more neural networksof deep neural network module 208, to reconstruct medical images frommeasurement data, according to one or more of the steps of method 300,shown in FIG. 3 below.

Non-transitory memory 206 may further store image data 214, such asmeasurement data or corresponding reconstructed medical images acquiredby imaging device 240. The medical images stored in image data 214 maycomprise MR images captured by an MRI system (in embodiments in whichimaging device 240 is an MRI imaging device), CT images captured by a CTimaging system (in embodiments in which imaging device 240 is a CTimaging device), and/or one or more types of measurement data/rawimaging data. For example, image data 214 may include measurement dataand corresponding ground truth medical images, which may be stored in anordered format, such that measurement data an anatomical region of asubject is associated with a ground truth medical image of the sameanatomical region of the same subject.

In some embodiments, the non-transitory memory 206 may includecomponents disposed at two or more devices, which may be remotelylocated and/or configured for coordinated processing. In someembodiments, one or more aspects of the non-transitory memory 206 mayinclude remotely-accessible networked storage devices configured in acloud computing configuration.

Medical imaging system 200 further includes imaging device 240, whichmay comprise substantially any type of medical imaging device, includingMRI, CT, PET, hybrid PET/MR, ultrasound, etc. Imaging device 240 mayacquire measurement data of an anatomical region of a patient, whereinthe measurement data may comprise a non-humanly intelligible encoding ofimaging data of the anatomical region. Measurement data, and medicalimages reconstructed therefrom, may be stored in image data 214, or inother non-transitory storage devices communicably coupled with medicalimaging system 200.

Medical imaging device 200 may further include user input device 230.User input device 230 may comprise one or more of a touchscreen, akeyboard, a mouse, a trackpad, a motion sensing camera, or other deviceconfigured to enable a user to interact with and manipulate data withinimage processing system 202.

Display device 220 may include one or more display devices utilizingvirtually any type of technology. In some embodiments, display device220 may comprise a computer monitor, and may display medical images, andmeasurement data. Display device 220 may be combined with processor 204,non-transitory memory 206, and/or user input device 230 in a sharedenclosure, or may be a peripheral display device and may comprise amonitor, touchscreen, projector, or other display device known in theart, which may enable a user to view medical images reconstructed frommeasurement data acquired by imaging device 240, and/or interact withvarious data stored in non-transitory memory 206.

It should be understood that medical imaging system 200 shown in FIG. 2is for illustration, not for limitation. Another appropriate medicalimaging system may include more, fewer, or different components.

Turning to FIG. 3 , an example method 300 for reconstructing medicalimages from measurement data using a deep neural networks and adecimation strategy is shown. Method 300 may be implemented by one ormore of the systems described above. In one embodiment, medical imagingsystem 200 may implement method 300 to reconstruct a medical imagecorresponding to measurement data acquired by imaging device 240.

Method 300 begins at operation 302, wherein the medical imaging systemreceives measurement data. In one embodiment, the medical imaging systemmay acquire medical imaging data of an anatomical region of a patientusing an imaging device, such as imaging device 240. In anotherembodiment, the medical imaging system may receive measurement data froma communicably coupled device, such as from an external imaging deviceconnected to the medical imaging device via the Internet, local areaconnection, or other digital connection. In some embodiments, themeasurement data received at operation 302 may comprise one of k-spacedata, CT sinogram data, PET sinogram data, optical imager andspectrometer (optical, infrared etc.) with Hadamard transform data asmeasurement data. Alternatively, in some embodiments, the measurementdata could be transformed to a linear transform space prior to operation304. In some embodiments, if the data is non-cartesian (e.g., radial orspiral trajectory), then the data may be re-gridded to a Cartesian gridbefore method 300 precedes to operation 304.

At operation 304, the medical imaging system selects a decimationstrategy. In one embodiment, a user may select a decimation strategyusing a user input device. In another embodiment, the medical imagingsystem may automatically select a decimation strategy based on one ormore properties of the measurement data. In one embodiment, a firstdecimation strategy may be selected in response to the measurement datacomprising 3D measurement data, and a second decimation strategy may beselected in response to the measurement data comprising 2D measurementdata. In one embodiment, the medical imaging system may select adecimation strategy based on a size of the measurement data, and apre-determined, non-zero, integer value, M, indicating an array size,wherein the decimation strategy may comprise decimating the measurementdata into a plurality of decimated measurement data arrays, each of sizeM. In some embodiments, imaging modalities which are represented byseparable linear transforms may be decimated using a same decimationstrategy.

At operation 306, the medical imaging system recursively decimates themeasurement data according to the decimation strategy selected atoperation 304, to produce a plurality of decimated measurement dataarrays. In some embodiments, the decimation strategy comprises adecimation factor, a sampling pattern, and a sampling frequency (whereinthe sampling frequency may be a function of the decimation factor). Insome embodiments, decimating the measurement data comprises sampling themeasurement data using the indicated sampling pattern and samplingfrequency, to produce a plurality of decimated measurement data arrays.In some embodiments, recursively decimating the measurement dataaccording to the decimation strategy comprises sampling a first fractionof data from the measurement data and assigning the first fraction ofdata to a first decimated measurement data array, and sampling a secondfraction of data from the measurement data and assigning the secondfraction of data to a second decimated measurement data array. In someembodiments, the first decimated measurement data array and the seconddecimated measurement data array may be decimated, to produce additionaldecimated measurement data arrays including less data, and/or lowerdimensional order data. This process may be continued until decimatedmeasurement data arrays of a pre-determined size are obtained.

In some embodiments, a user may select a size, M, of the decimatedmeasurement data arrays, and operation 306 may include decimating themeasurement data to produce a plurality of arrays of size M, wherein Mcomprises an integer greater than one. In one example, measurement datamay comprise a 100 row vector, and in response to a user selecting adecimated measurement data array size of 20, the medical imaging systemmay set a decimation factor of 5, and a sampling frequency of 0.20, andmay decimate the 100 row vector to produce five decimated measurementdata arrays, each comprising 20 rows.

At operation 308, the medical imaging system maps the plurality ofdecimated measurement data arrays to a plurality of decimated image dataarrays using a plurality of trained deep neural networks. In oneembodiment, mapping the plurality of decimated measurement data arraysto a plurality of decimated image data arrays comprises passing theplurality of decimated measurement data arrays through one or more fullyconnected/dense layers of one or more of the deep neural networks,wherein the one or more fully connected/dense layers of the one or moredeep neural networks comprise one or more learned filters, learnedduring a training phase, as discussed below in more detail, withreference to FIG. 4 .

At operation 310, the medical imaging system recursively aggregates theplurality of decimated image data arrays using one or more trainedaggregation networks to produce an image data array. In someembodiments, aggregating the plurality of decimated image data arrayscomprises inputting each of the plurality of decimated image data arraysinto an input layer of an aggregation network. The aggregation networkmay then apply one or more one-dimensional (1D) convolutional filters tothe plurality of decimated image data arrays to produce an image dataarray, wherein the image data array has a higher resolution thanplurality of decimated image data arrays. In some embodiments, pairs ofdecimated image data arrays may be aggregated using an aggregationnetwork, wherein a plurality of decimated image data arrays may beaggregated pairwise into larger and larger sub-units of an image dataarray. In an illustrative example, a first, second, third, and fourthdecimated image data array may be aggregated by inputting the first andsecond decimated image data arrays into an aggregation network, whereinthe aggregation network outputs a first partially aggregated image dataarray (also referred to herein as an image data array subunit).Similarly, the third and fourth decimated image data arrays may be inputinto the aggregation network, and mapped to a second partiallyaggregated image data array. The first and second partially aggregatedimage data arrays may then be input into the aggregation network, andmapped to the image data array. In the preceding example, the resolutionof the first and second partially aggregated image data arrays is largerthan the resolution of the first, second, third, or fourth decimatedimage data arrays, but is less than the resolution of the image dataarray. Aggregation of two or more decimated image data arrays may, insome embodiments, comprise aligning the decimated image data arrayvectors, and concatenating the data within the two or more decimatedimage data arrays by channel/row, to produce a channel-wise concatenateddecimated image data array. The channel-wise concatenated decimatedimage data array may then be input into an input layer of an aggregationnetwork, thereby preserving spatial information and providing theaggregation network with the spatial relationships between the intensityvalues of the two or more decimated image data arrays.

At operation 312, the medical imaging device reshapes the image dataarray to produce a reconstructed medical image. In some embodiments,reshaping the image data array to form the reconstructed medical imagecomprises re-arranging the rows of the image data array into a matrix,wherein each column of the matrix consists of an equal number of rows.In one example, operation 312 may include dividing the image data arrayat a plurality of pre-determined rows, to produce a plurality ofcolumns, and re-arranging the columns according to a pre-determinedorder, thereby producing a matrix, depicting the image data of imagedata array in a humanly recognizable format.

At operation 314, the medical imaging device displays the reconstructedimage to a user, via a display device.

Following operation 314, method 300 may end. In this way, method 300 mayenable more computationally efficient medical image reconstruction,without sacrificing image resolution, and without increasing an amountof noise in the reconstructed image, compared to conventionalapproaches.

A technical effect of decimating measurement data to form one or moredecimated measurement data arrays, is that a computational complexity ofmapping the measurement data to image data may be reduced from O(N⁴),where N is the size of the measurement data, to O(M⁴), where M is thesize of an individual decimated measurement data array, wherein M<N.

Turning to FIG. 4 , an example of a training method 400, which may beexecuted by one or more of the systems described above, is shown. In oneembodiment, method 400 may be used to train a medical imagereconstruction system, such as medical image reconstruction system 100,shown in FIG. 1 . In some embodiments, training method 400 may be usedto train one or more of the deep neural networks used in method 300 toreconstruct a medical image from measurement data. In one embodiment,training method 400 may be used to learn one or more convolutionalfilters used in one or more of a plurality of convolutional neuralnetworks used to map decimated measurement data arrays to decimatedimage data arrays. Method 400 enables training of both the convolutionalneural networks, and the aggregation networks of a medical imagereconstruction system, at substantially the same time, using a same setof training data, thereby increasing an efficiency of the trainingprocess.

Method 400 begins at operation 402, where the medical imaging systemselects a training data pair comprising measurement data and acorresponding ground truth medical image, based on the current medicalimage reconstruction system being trained. In one example, a trainingdata pair may be selected from a repository of training data, such asimage data 214 of medical imaging system 200, based on metadataassociated therewith. In some embodiments, for each imaging modality, adistinct medical image reconstruction system may be trained, therebyenabling the deep neural networks and aggregation networks of a givenmedical image reconstruction system to learn the manifolds relatingmeasurement data in a particular imaging modality, to image data. Inother words, the performance of a medical image reconstruction systemmay be enhanced by selectively training the medical image reconstructionsystem using training data comprising measurement data and acorresponding ground truth medical image, of a single type of imagingmodality. Therefore, at operation 402, the medical imaging system mayselect a training data pair based on a pre-determined imaging modalityfor which the current image reconstruction system is to be trained.Metadata associated with the training data pair may indicate a type ofimaging modality to which the training data pair is associated.

At operation 404, the medical imaging system selects a pre-determineddecimation strategy based on the current medical image reconstructionsystem being trained. In some embodiments, for each decimation strategy,a distinct medical image reconstruction system may be trained, therebyenabling the deep neural networks and aggregation networks of a givenmedical image reconstruction system to learn a mapping from a particularsize of decimated measurement data arrays to image data. In other words,the performance of a medical image reconstruction system may be enhancedby selectively training the medical image reconstruction system using asingle decimation strategy. Therefore, at operation 404, the medicalimaging system may select a decimation strategy based on the currentmedical image reconstruction system being trained, wherein the medicalimage reconstruction may include an indication of an associateddecimation strategy for which the deep neural networks of the medicalimage reconstruction system are compatible.

At operation 406, the medical imaging system recursively decimates themeasurement data according to the selected decimation strategy toproduce a plurality of decimated measurement data arrays. In someembodiments, recursively decimating the measurement data according tothe selected decimation strategy may include setting a decimationfactor, a sampling pattern, and a sampling density (which may be afunction of the decimation factor) according to the decimation strategyselected at operation 404, and producing a number of decimatedmeasurement data arrays thereby. Decimation is discussed in more detailabove, with reference to decimation 108 of FIG. 1 .

At operation 408, the medical imaging system maps the plurality ofdecimated measurement data arrays to a plurality of decimated image dataarrays using a plurality of deep neural networks of the current medicalimage reconstruction system, wherein the plurality of deep neuralnetworks comprises one or more un-trained convolutional neural networks.

At operation 410, the medical imaging system recursively aggregates theplurality of decimated image data arrays using one or more aggregationnetworks to produce a predicted image data array, as described in moredetail above, with reference to operation 310 of FIG. 3 .

At operation 412, the medical imaging system reshapes the predictedimage data array to produce a predicted medical image, as discussed inmore detail with reference to operation 312 of FIG. 3 , above.

At operation 414, the medical imaging system calculates an error of thepredicted medical image by comparing the predicted medical image and theground truth medical image using a loss function. In one embodiment,pixelwise error may be determined by inputting a pixel intensity (forone or more colors) of the predicted image and pixel intensity (for oneor more colors) of a corresponding pixel of the ground truth medicalimage, into a loss function. Each pixelwise error may be summed, toproduce a total error (herein more simply referred to as the error) ofthe predicted medical image.

At operation 416, the medical imaging system adjusts parameters of theone or more aggregation networks and the plurality of deep neuralnetworks by backpropagating the error through the one or moreaggregation networks and the plurality of deep neural networks using abackpropagation algorithm. In one embodiment, operation 416 comprisesthe image processing system adjusting the weights and biases of the oneor more aggregation networks and the plurality of deep neural networksbased on the error calculated at operation 414. In some embodiments,back propagation of the error may occur according to a gradient descentalgorithm, wherein a gradient of the error function (a first derivative,or approximation of the first derivative) is determined for each weightand bias of the plurality of deep neural networks and the one or moreaggregation networks. Each weight (and bias) of the plurality of deepneural networks and the one or more aggregation networks is then updatedby adding the negative of the product of the gradient determined (orapproximated) for the weight (or bias) and a predetermined step size,according to the below equation:

$P_{i + 1} = {P_{i} - {\eta\frac{\partial{Error}}{\partial P_{i}}}}$

where P_(i+1) is the updated parameter value, P_(i) is the previousparameter value, η is the step size, and

$\frac{\partial{Error}}{\partial P_{i}}$is the partial derivative of the error with respect to the previousparameter.

Following operation 416, method 400 may end. It will be appreciated thatmethod 400 may be repeated until one or more conditions are met. In someembodiments, the one or more conditions may include the weights andbiases of the one or more aggregation networks and the plurality of deepneural networks converging (that is, a rate of change of the parametersof the plurality of deep neural networks and one or more aggregationnetworks decreases to below a pre-determined threshold rate), the errordetermined at operation 414 decreases to below a pre-determined,non-zero, threshold (in some examples, the error may be determined usinga validation dataset, distinct from the training data set).

In this way, both the plurality of deep neural networks and the one ormore aggregation networks may be trained using a joint-training method,wherein a single training data set may be used to adjust parameters(e.g., filters/weights and biases) of both the plurality of deep neuralnetworks and the one or more aggregation networks, increasing a trainingefficiency and reducing a duration of time required to train the medicalimage reconstruction system.

Turning to FIG. 5 , a comparison between a medical image reconstructedusing the approach of the current disclosure (first medical image 502)is compared against a medical image reconstructed using a conventional,and computationally expensive, reconstruction approach (second medicalimage 504). As can be seen, the first medical image 502 closely matchesthe anatomical structures included in the second medical image 504, andprovides a comparable level of resolution.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” and “the” are intended to mean thatthere are one or more of the elements. The terms “first,” “second,” andthe like, do not denote any order, quantity, or importance, but ratherare used to distinguish one element from another. The terms“comprising,” “including,” and “having” are intended to be inclusive andmean that there may be additional elements other than the listedelements. As the terms “connected to,” “coupled to,” etc. are usedherein, one object (e.g., a material, element, structure, member, etc.)can be connected to or coupled to another object regardless of whetherthe one object is directly connected or coupled to the other object orwhether there are one or more intervening objects between the one objectand the other object. In addition, it should be understood thatreferences to “one embodiment” or “an embodiment” of the presentdisclosure are not intended to be interpreted as excluding the existenceof additional embodiments that also incorporate the recited features.

In addition to any previously indicated modification, numerous othervariations and alternative arrangements may be devised by those skilledin the art without departing from the spirit and scope of thisdescription, and appended claims are intended to cover suchmodifications and arrangements. Thus, while the information has beendescribed above with particularity and detail in connection with what ispresently deemed to be the most practical and preferred aspects, it willbe apparent to those of ordinary skill in the art that numerousmodifications, including, but not limited to, form, function, manner ofoperation and use may be made without departing from the principles andconcepts set forth herein. Also, as used herein, the examples andembodiments, in all respects, are meant to be illustrative only andshould not be construed to be limiting in any manner.

The invention claimed is:
 1. A method comprising: receiving measurementdata acquired by an imaging device; flattening the measurement data;selecting a decimation strategy based on a size of the measurement data;producing a reconstructed image from the flattened measurement datausing the decimation strategy and a plurality deep neural networks,where the selection is further based a size of an input layer of theplurality of deep neural networks; decimating the measurement dataaccording to the decimation strategy to produce at least a firstdecimated measurement data array and a second decimated measurement dataarray; mapping the first decimated measurement data array to a firstdecimated image data array using a first deep neural network; mappingthe second decimated measurement data array to a second decimated imagedata array using a second deep neural network; aggregating the firstdecimated image data array and the second decimated image data arrayusing an aggregation network to produce an image data array; reshapingthe image data array to produce the reconstructed image; and displayingthe reconstructed image via a display device.
 2. The method of claim 1,wherein the measurement data acquired by the measurement devicecomprises tomographic data related to the reconstructed image via asystem transform.
 3. The method of claim 1, wherein the measurement datacomprises imaging measurement data, and wherein the imaging devicecomprises one of an MRI scanner, a CT scanner, a PET scanner, a PET/MRscanner, an optical scanner with Hadamard imaging, an optical andinfrared Hadamard spectrometer, and a C-Arm scanner.
 4. The method ofclaim 1, wherein the aggregation network comprises a one-dimensional(1D) convolutional layer.
 5. The method of claim 1, wherein the firstdeep neural network comprises a first fully connected layer, and whereinthe second deep neural network comprises a second fully connected layer.6. The method of claim 1, wherein the decimation strategy includes adecimation factor and a sampling pattern.
 7. The method of claim 6,wherein a size of each of the first decimated measurement data array andthe second decimated measurement data array is determined based on thedecimation factor.
 8. The method of claim 1, wherein the measurementdata is first measurement data, and wherein the first deep neuralnetwork, the second deep neural network, and the aggregation network,are trained by: selecting a training data pair comprising secondmeasurement data and a corresponding ground truth image; decimating thesecond measurement data according to the decimation strategy to producea plurality of decimated measurement data arrays; mapping the pluralityof decimated measurement data arrays to a plurality of decimated imagedata arrays using the first deep neural network and the second deepneural network; aggregating the plurality of decimated image data arraysusing the aggregation network to produce a predicted image data array;reshaping the predicted image data array to produce a predicted image;calculating an error of the predicted image by comparing the predictedimage and the ground truth image using a loss function; and adjustingparameters of the first deep neural network, the second deep neuralnetwork, and the aggregation network, based on the error.
 9. The methodof claim 8, wherein adjusting parameters of the first deep neuralnetwork, the second deep neural network, and the aggregation networkbased on the error comprises: backpropagating the error through eachlayer of the aggregation network, the first deep neural network, and thesecond deep neural network; and adjusting parameters in each layer ofthe aggregation network, the first deep neural network, and the seconddeep neural network, to reduce the error.
 10. An imaging system,comprising: a display device; an imaging device; a memory storing aplurality of trained deep neural networks, one or more aggregationnetworks and instructions; and a processor communicably coupled to thedisplay device, the imaging device, and the memory, and when executingthe instructions, configured to: acquire measurement data of ananatomical region of a patient using the imaging device; flattening themeasurement data; select a decimation strategy; recursively decimate theflattened measurement data according to the decimation strategy to forma plurality of decimated measurement data arrays; map the plurality ofdecimated measurement data arrays to a plurality of decimated image dataarrays using the plurality of trained deep neural networks; andrecursively aggregate the plurality of decimated image data arrays,using a plurality of trained aggregation networks, to produce an imagedata array; reshaping the image data array to produce a reconstructedimage; and display the reconstructed image via the display device. 11.The imaging system of claim 10, wherein the imaging device is one of anMRI scanner, a CT scanner, a PET scanner, a PET/MR scanner, an opticalscanner with Hadamard imaging, an optical and infrared Hadamardspectrometer, and a C-Arm scanner.
 12. The imaging system of claim 11,wherein the medical imaging device is the MRI scanner, and themeasurement data comprises k-space data of the anatomical region of thepatient acquired by the MRI scanner.
 13. The imaging system of claim 10,wherein the processor is configured to recursively decimate themeasurement data according to the decimation strategy to form theplurality of decimated measurement data arrays by sampling a firstfraction of data from the measurement data and assigning the firstfraction of data to a first decimated measurement data array, andsampling a second fraction of data and assigning the second fraction ofdata to a second decimated measurement data array.